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ABSTRACT 

Seven schools in which pupil achievement was high 
were matched with seven schools in which pupil achievement was low. 
Nine teachers were selected for observation in each school; teachers 
filled out questionnaires, and principals were interviewed. 
Significant differences in favor of schools with high pupil 
achievement were found in teacher expectations for pupils, in teacher 
attitudes toward pupils ^ in the use of positive reinforcement, in the 
cognitive load of materials presented, and in a number of related 
variables. Principals in schools with high pupil achievement 
generally saw their teachers as more competent. Implications of these 
findings for further work on variables of teacher behavior are 
discussed, (Author) 
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Introduction 

This study represents the third phase of a research effort which 
began in 1970, David Irvine and John Heim (1973)? Bureau of School 
Programs Evaluation, New York State Education Department, noted a wide 
range of test scores among several New York City scools even thoiogh all 
of the schools drew from low socio-economic populations. High and low 
scoring schools from the Irvine and Heim study were identified and 
labeled as "outlyers»" 

As a next step, Alan Robertson, Director of the Division of 
Education Evaluation, found that staff members of the New York Central 
Board of Education, who recognized the schools were in every case able 
to identify the school as high or low. Because the high and low 
schools did appear to be viewed differently by people who knew them, 
an effort \^s made to determine the extent to which school environ- 
ment, school program, and administrative conditions might be related 
to the different test scores. 

Next an effort was made to determine whether schools dodged to be 
high or low performers in the Irvine-Heim study were consistently high 
or lov7, or whether this was an accidental finding which Just happened 
to occur in one school year. A statistical analysis to determine con- 
sistency of school building reading scores was performed. Identified 
were a group of five schools whose reading scores were consistently 
high and five schools in which scores were consistently low. In 
addition, a school was found in which test scores increased over a two 
year period, and a school was found in which scores declined over the 
same two year period • 



The twelve schools identified were then used as the targets for 
structured observations, "Blind" evaluators classified eight out of 
ten of the high and low schools correctly. In seven out of ten cases, 
all evaluators judged correctly. (Generally, three evaluators visited 
each school.) While specific differences between high and low scores 
were not completely consistent, seven factors emerged as likely to be 
more true of high achieving schools than of low achieving schools. 
Briefly, these characteristics are: 

1, Teachers manifested better rapport with students. 

2, Teachers exercised more effective control of pupils. 

3, Teachers engaged in more extensive preparation of lessons. 
k. Reading instruction was at a more appropriate level for the 

needs of pupils. 

5. Teachers provided for more extensive regrouping within the 
reading period, 

6. Teachers provided, for more extensive use of material in the 
reading program. 

7. The soin-ces of leadership in instruction in reading were more 
forceful and positive. 

Because certain behaviors did indeed seem to distinguish high and 
low achieving schools, it was suggested that, "With the elimination or 
control of some of the design problems enumerated earlier, and with 
more extended observations in the schools, it would be extremely valu- 
able to see whether these same behavioral variables emerge in distin- 
guishing high and low achieving schools." 

As a follow-up to this recommendation a group of faculty and 
students at the State University of New York at Albany were asked to 
revise the instruments used in New York City, to design new instru- 
ments that might be used to measure variations in teaching, and to 
employ these instruments in selected schools. Fourteen schools were to 
be selected by the State Education Department. Seven of the selected 
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schools showed consistently high performance and seven relatively low 
performance • 

Overview of the Instrument Develoianent Process 

As a first step, the original forms used in the Outlyer Study were 
examined and discussed by the group. From the original Outlyer forms, 
a list -was rnade of items for further consideration. This list was dis- 
cussed by the group in terms of the importance of each item and the 
psychometric problems connected with each item. In addition, consider- 
ation was given to ways and means in which relevant information might 
be gathered in the school. Specific scales and general scale defini- 
tions were then developed. 

Arrangements were made for field tryouts in two public schools. 
In each case, foiir observers visited the school. Pairs of observers 
made twenty-minute observations in six different classrooms so that 
each observer was paired with every other observer. Also a pair of 
observers interviewed the reading teacher and the other two observers 
interviewed the school principal. 

After the first field trials had been made, data were inspected 
for reliability, and a group meeting was held to discuss problems and 
procedures. In this meeting agreement was reached that some forms 
needed revision, some categories needed redefinition, and some new 
areas needed to be included. Again, individuals took responsibility 
for revision. Revisions were made, ftirther editing was done, and 
forms were prepared for use in fourteen schools to be selected by the 
State Education Department. 
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Seven of the instruments developed are discussed in this paper. 
Pour of these instruments were designed to be used in each of the class- 
rooms visited^ Also developed were a principal interview schedule and 
a reading teacher interview schedule, and a questionnaire to be filled 
out by the teachers observed. 

Procedin'es for School Visits 

After instruments had been revised and constructs redefined, foxir- 
teen school buildings in upstate New York were selected. Seven build- 
ings were nominated because PEP reading scores were relatively high and 
seven buildings because scores were relatively low. Individuals doing 
the observations were imaware of whether the school which they visited 
was "high" or "low." 

For each school visit the principal was asked to arrange access 
to nine elementary school classrooms between kindergarten and sixth 
grade. (In a few of the smaller schools, nine classrooms x-zere not 
available.) Each observer was assigned to four of these classrooms, 
and the pair of observers were together in one classroom dtiring the day. 
In addition, an interviev/ was held with the school principal and with a 
reading teacher in the school. 

Analysis of Data - Overview 

In this report a brief description is provided of each of the 
seven instilments that were developed, and means are provided for 
classes in the high and low schools. In some cases separate analyses 
are provided for grades 1-3 and grades U-6. T-tests of significance 
of the differences between the means were computed except for principal 
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interview and reading teacher interviews when the N for each group was 
only seven. 

General Classroom Observation Form 

The General Classroom Observation Form consists of l6 items. Ques- 
tions are grouped under the areas of program emphasis, teacher behavior, 
pupil behavior, and facilities, A five-point Likert scale was used. 
Operational definitions were developed for each end of the scale. 

Data from the General Classroom Observation Form reported in Table 
1 shox^r clear differences between classes in high and lov7 schools in 
grades 1-3 but relatively little difference between classes in grades 

Table 1 about here 

For early elementary grades, items which did not differentiate the 
groups were program emphasis on social development, rigidity of student 
behavior, and the three items on facilities. Classes in high reading 
schools were significantly higher on all items except "effort to main- 
tain control," and "rigidity of student behavior." For these ratings, 
high reading classes in grades 1-3 were significantly lower. Thus, in 
grades 1-3 classes in high reading schools, teachers don^t work as hard 
to maintain control and have less rigid student behavior, but at the 
same time are rated significantly higher in effectiveness of control. 
Also teachers in high reading schools are rated as warmer, more suppor- 
tive, more responsive to students, and showing more emphasis on cogni- 
tive development. Pupils in their classes appear more enthusiastic 
about school and better able to sustain attention. 
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In grades li.-6, however, only the differences in teacher efforts to 
maintain control and effectiveness of control were significant. The 
pattern was the same as in the lower grades, with teachers in high 
reading schools making less effort to maintain control but being rated 
more effective in control, 

0bserv8.tion of a Reading Group 

The form used for observation of reading classes is a modification 
of an observation system developed and tested by Educational Testing 
service, Quirk, et. al. 1973, Weinberg, et. al. 197^. To develop the 
original instrument, members of a research team visited second, fourth, 
and sixth grade reading classes and kept a log of the activities that 
took place. Eventually, they arrived at twelve categories to describe 
what they called the Content of Instruction. They also developed de- 
finitions and examples of each area. The procedure used by ETS called 
for a different student to be scored on the instrument dxiring each 
fifteen second interval in an observation period of fifteen minutes. 

Because in this project a broader range of observational data were 
desired, it was decided to attempt to score each category of Content of 
Instruction on a Likert scale arranged from "Little" to "Much." The 
categories "Extraneous" and "Negative Feedback" were dropped from the 
scale since other observation devices covered these areas. Separate 
ratings were to be made of the activities of children in a reading 
group directed by the teacher and children not in the reading group. 

Thus, the form used in the first field tryouts consisted of ten 
categories to be rated for the reading group and the children not in 
the reading group. Prom these tryouts it wa.s determined that one 
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additional area^ oral reading^ was needed for the reading group. A 
definition was written for this area. Also, observers found that the 
categories available did not allow adequate description of the behavior 
of children not in reading groups, and that too much inference was re- 
quired to determine whether a child writing at his seat vms worldLng on 
word recognition, language structure, or spelling. Therefore, the list 
of items to be rated for the non-reading group was revised and new 
definitions \fritten. 

a?he Observation of Reading form was designed to be used only when 
the teacher was engaged in direct reading instruction. When arrange- 
ments for school visits v;ere made it was stressed that some reading 
classes would be observed, but classes would also be visited diiring 
other activities. Since observers were in classes most of the school 
day, and reading was normally scheduled in the morning, reading classes 
were often not available. In Table 2 are presented the mean ratings 
for activities carried on in reading classes in high and low schools. 

Table 2 about here 

As can be seen, relatively few reading classes were observed in grades 
four to six. 

In grades one to three more total activity seemed to occtir in the 
high schools. In nine of the eleven activities rated, more emphasis 
was given in high schools than in low schools. The two areas in which 
higher means were recorded for low schools were for reading orally and 
for management instructions. The greatest difference between high and 
low schools was in the relatively large amount of silent reading going 
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on in high schools. Thus, in classes in high schools children were more 
likely to read silently, and in low schools to read out loud. Althou^ 
the number of classrooms is very small, this relationship between silent 
and oral reading extends to grades four to six. 

Also examined in the Observation of Reading form were the behaviors 
of children who were not in reading groups. These data are also re- 
ported in Table 2. 

Teacher Reinforcement Scale 

The Teacher Reinforcement Scale used in this set of observations 
was developed for use in this study. From the review of literatxire it 
seemed clear that teacher reinforcement might well be a critical vari- 
able in how children learn, but the problem remained of how to score 
this domain in simple but meaningftil way. It was agreed that one might 
discriminate between positive reinforcement in the form of praises or 
token reward, and punishment in the form of scolding, criticizing, 
withholding privileges, and the like. These punishing behaviors are 
labeled as "negative reinforcement" on the form, although they do not 
fit the classical learning definitions of negative reinforcement. 

Along with the distinction of positive and negative, it was agreed 
that a distinction needed to be made between frequency and potency. 
Some teachers used a great number of remarks such as "good" or "cor- 
rect," but they used these remarks so routinely that observers 
wondered if they x^rould really have much effect. Other teachers did 
not praise as frequently, but they extended and elaborated their com- 
ments. Obviously, an observer can't be sure of the effect of either 
form of comment on a child, but the rating of potency is included as a 
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subdective measiire in which the observer attempts to score the meaning- 
fulness of the reinforcement given* 

Finally, it \ms agreed that what the teacher reinforced ims of 
interest. In some cases teachers praise or punish the child's actual 
product — his math paper, or the way he reads. At other times teachers 
praise or punish the child's general conduct or social behavior — the 
way he pays attention or works on an assignment. Therefore, it vms 
decided to attempt to separate ratings for instructional specific and 
general support/social behavior reinforcement. 

Data are reported in Table 3, As can be seen, teachers in grades 

Table 3 about here 

one to three in high schools provided more positive reinforcement tinder 
all conditions, and teachers in low schools provided more negative re- 
inforcement. However, in grades U-6 teachers in low schools provided 
more reinforcement, both positive and negative, than did teachers in 
high schools. 

In the instructional specific category teachers in all groups were 
recorded on the average providing more positive reinforcement than nega- 
tive reinforcement. Under the category of general support, however, 
teachers in high schools had a higher frequency of positive than nega- 
tive reinforcement, while teachers in low schools showed the reverse 
pattern. As the observers scored these categories, only two mean 
ratings reached the midpoint of the five-point scale. In general, the 
observers did not record high amounts of reinforcement. 
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C haracteristics of Open Education 

Items from the Characteristics of Open Education form were derived 
from the Walberg-Thomas (l970) instrument. In their procedure, teacher 
intervievj was used as a means of supplementing classroom observation, 
while in this study only observation was used to derive data. Since 
this form ms not used in preliminary field work, no data were avail- 
able to form a basis for revision. Observers reported that some items 
were not really suitable for observation and also indicated that 
clearer polar definitions would be useful o 

However, in spite of these problems, a definite trend is sson in 
higher mean scores of classes in high schools, (Higher scores ireaning 
more open.) Of the l8 comparisons of classes in grades 1-3^ 15 are in 

Table h about he e 

favor of high classes, as can be seen in Table k. This trend was not 
strong in classes of grades in which means for classes in high 
schools were greater in 11 of eighteen comparisons. For almost all 
items of open education characteristics, ratings were belo^f the mid- 
point of the five-point scale. Thus, one might assume that the general 
sample of schools employed in this study were relatively traditional, 
and this assumption is verified by the opinions of the observers. 

Teacher Questionnaire 

In the first planning meeting the Project team discussed the ques- 
tion of how to obtain information from teachers that could not be 
gained through observation. Clearly, it would be desirable to loiow 
about teacher intentions, philosophy, evaluations of children, 
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relationships with administration^ and a host of other variables. At 
the same time, to build in even a modest teacher interview would create 
great scheduling problems and woixld drastically reduce the number of 
teachers who could be observed during a school visit. 

The decision was finally reached to prepare a questionnaire which 
could be responded to very quickly. The areas of concern in this ques- 
tionnaire were derived primarily from variables that appeared promising 
in the Outlyer Study. Thus, the teacher is asked about her expectancies 
for the children she teaches, her assessment of the general ability and 
attitude of her present class, the degree to which she would expect 
help for various problems that might be encountered, and her assessment 
of the locus of control for decision-making. 

Teacher questionnaires were left in each of the schools visited, 
and a request was made to retiirn the questionnaires, when completed, 
to the Project Director. While some questionnaires were not retiirned, 
a sample of kQ responses from high schools and 51 responses from low 
schools were available for analysis. The first four items of the ques- 
tionnaire concerned teacher expectations about children in their class 
projected into the future. Teachers answered these questions in terms 
of percentages. These data are reported in Table 5. As can be seen. 

Table 5 about here 

teachers in high schools had higher expectations for children in all 
foTir of the areas questioned. It is interesting to note in each group 
that teachers expect more children to graduate from high school than 
they expect to become fluent readers. 



ERLC 



.3 



A similar pattern is seen when teachers were asked about percep- 
tions of their children in the here and now. Teachers in high schools 
see children in their classes as more intelligent, better behaved, more 
pleasant to teach, and having more concerned parents. 

Also derived from the teacher questionnaire were teacher expecta- 
tions for various kinds of support within the school. These data are 
also provided in Table 5, As can be seen, responses of teachers in 
high and low schools are very close. In terms of these data, at least, 
difference in administrative support as perceived by teachers is not a 
critical factor. 

Principal Interview 

The Principal Interview form was developed in large part from vari- 
ables that seemed to be of interest from the original Outlyer Study. 
(Irvine and Heira, 1973) An effort was made to obtain (l) specific de- 
mographic information about the school; (2) subjective impressions of 
the principal concerning the professional staff in the school, the ade- 
quacy of facilities and material support for the reading program, the 
locus of control of the reading program; and (3) principal's judgement 
of special problems and special assets of the school. 

Although the form calls for precise answers to most of the ques- 
tions, the procedure specifies an informal interview approach. Thus, 
the person obtaining data was encouraged to engage the principal in 
discussion of the areas to be covered and to probe in specific areas 
until a scorable answer was obtained. The order of questions might be 
modified to fit the circumstances. 



Data are reported in Table 6. As can be seen, principals in high 
schools provide consistently higher ratings than do principals in low 
schools. All average ratings fall above the midpoint of the Likert 
scale, indicating that, on the average, all principals gave a favorable 
rating. 

Table 6 about here 

Principals were also asked a set of items about their own rapport 
with various groups with whom they deal. These data are also summarized 
in Table 6. Principals in low schools were rated from the interview as 
higher in rapport with the board, but lower in rapport with teachers, 
parents, and children. Especially noticeable is the difference of the 
two groups in ratings for rapport with children. However, because the 
number of principals in each group is only seven, conclusions must be 
tentative. 

Reading Teacher Interview 

The interview guide used with the reading teacher in each school 
was adopted from an "Observer Guide-Reading" which was made available 
to the Project Director by Mrs. Jane Algozzine, C!hief of the Bureau of 
Reading, State Education Department. Originally, this observation 
instrument was used in direct classroom observation, supplemented by 
teacher interview, to describe the degree to which reading practices 
seen as ideal were actually practiced. The original instrument con- 
sisted of 13 categories, each to be evaluated from "low" to "high" on 
a five-point scale. Paragraphs describing "lo\/" and "high" practices 
were provided for each item. Also, in the original instrument, con- 
siderable space was provided for comments on each room. 



In this study it was decided to use ten of the thirteen "Observer 
Guide-Reading" categories in an interview format with the reading 
teacher. The definitions of "low'' and "high" behavior for these ten 
categories were used as a guide for interviewer scoring. Aspects of 
the reading program which were questioned related primarily to reading 
as it is carried on in the classroom. Therefore, the reading person 
who seldom visits the classroom or talks with the teacher could not be 
expected to give valid responses to the questions asked. However, all 
reading teachers did indicate a general familiarity of the reading pro- 
gram as carried on in the classroom and did seem to feel that they had 
a good idea of the answers to these questions about the reading program. 

Data are reported in Table 7. It should be noted that the averages 
in the table are compiled on the responses of the seven reading teachers 

Table 7 about here 

in the high schools and the seven in the low schools. Althoiogh the 
sample size is small, a definite trend can still be seen since higher 
means are recorded for the high schools in nine of the ten contrasts. 
Only on item six, which concerns the degree to which reading material 
matches the ethnic background of the child, were low schools given a 
higher mean. 

Also of interest are the absolute scores assigned from the five- 
point Likert scale. In general, absolute scores are higher than the 

i 

scores assigned through direct classroom observation. It is not sur- 
prising that reading teachers should generally see the program in their 
school in a favorable light. Nevertheless, especially in the low 
school, most ratings fall between three and 3.5, suggesting that reading 
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teachers see rocjm for improvement, (improvemsnt, assmiing behaviors on 
the high end of the scale are judged superior to behaviors on the low 
end.) 

Summary and Conclusions 

At the end of the study each observer was asked to submit in -writ- 
ing his impressions of each of the devices , problems they faced and the 
like. There was general agreement that some items on the Characteris- 
tics of Open Education form needed to be dropped, since they cannot be 
observed. Also, some observers questioned whether the reading teacher 
really knew enough about what was actually happening in classrooms to 
be a valid soiirce of data. Generally it was agreed that evaluation of 
teacher reinforcement ^Tas important but very difficult. Especially at 
the end of the year it was felt that much reinforcement might be very 
subtle and hard to observe. 

A limitation of this study, noted by several observers, was the 
fact that the work was carried out very near the end of the school 
year. Of great interest is the question of whether similar differences 
would be noted at the beginning of the year. Perhaps, in the area of 
management instruction for example, some teachers give much direction 
very early in the year, establish a firm routine, and need to give re- 
latively few such directions thereafter. Observers in this study re« 
ported more management instruction in low than in high schools, but it 
would be interesting to note whether this difference is the same at the 
beginning of the school year. 

Thus, next steps that might be taken in this area are: 
(1) Select the variables that seem to be related to the clearest 
differences between schools. 
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(2) Attempt to clarify further the behaviors that are being rated 
and the criteria for rating each of these variables. 

(3) Consider whether the variables identified by these procedures 
could be meaningfully divided into sub-parts to be more speci- 
fically studied, 

{h) Try out revised materials on a broader geographic basis and 
with schools that are demographically more diverse. 

(5) Experiment with these materials in an in-service and/or pre- 
service context. 

(6) Work systematically with a group of teachers to see if teachers 
can learn to vary selected behaviors, and study the effects of 
such variations • 

Summary of Most Significant Findings 

1. Teachers in high schools made less overt effort to maintain class 
control, had less rigid student behavior but were more effective 
in maintaining the level of control they appeared to want. 

2. Teachers in high schools were rated as warmer, more responsive, 
and showing more emphasis on cognitive development in classes that 
did not involve direct reading instruction as well as in reading 
classes. 

3. Teachers in high schools expected more children to graduate from 
high school, to go to college, to become good readers and to be- 
come good citizens. 

If.. Teachers in high schools see their children as more intelligent, 
better behaved, more pleasant to teach, and their parents as more 
concerned. 
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5. Teachers in high and low schools do not see different amounts of 
help available in handling problems, 

6. More total activity takes place in reading classes in high schools 
than in low schools. 

7. Children in reading classes in high schools engage in more silent 
reading while children in low schools engage in more oral reading • 

8. Reading teachers in evaliiating the classroom reading program were 
on the average more favorable in high schools. Reading teachers 
rated teachers in high schools more favorably in using appropriate 
material, extending reading into other areas, asking children to 
read with a purpose, and using informal diagnosis. 

9. In grades one to three teachers in high schools gave more positive 
and less negative reinforcement than did teachers in low schools. 

10. In grades four to six teachers in low schools gave more reinforce- 
ment. In general, however, teachers in high schools gave positive 
reinforcement more than negative. 

11 o On selected items related to open education high schools appeared 
more open than low schools. 

12. On several of the measures, differences between high and low 
schools seemed more pronounced in grades one to three than in grades 
four to six. 

13. Of three devices tested, two devices significantly differentiated 
high and low schools arranged on a rank order basis. Thus, not 
only were means different, but total schools were also different. 

ik. In general, reasonably close relationships were found between the 
ratings of different observers. 
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15. Principals in high schools generally saw their personnel as more 
competent than did principals in low schools. 

16. Principals in high schools saw themselves as having better rapport 
with teachers 3 parents and pupils than did principals in low schools. 
Principals in low schools reported better rapport with the school 
board. 

17. Items on physical space and facilities generally did not differen- 
tiate between high and low schools. 
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